DOCOHEMT BSSOHE 



BD 101 173 

AUTHOB 
TITLE 

INSTITUTION 



PUB DATE 
NOTE 



EDBS PBICE 
DESCBIPTOBS 



CB 002 9H2 

Stephenson, Bobert H. ; And Others 
A Nodular Approach to Proficiency Testing. 
American Institutes for Pesearch in the Behavioral 
Sciences, Washington, D.C.; Army Research Inst, for 
the Behavioral and Social Sciences, Arlington, Va. 
Sep 73 

29p. ; Paper presented at the Annual Conference of the 
Military Testing Association (15th, San Antonio, 
Texas, October 28-Noveiber 2, 1973) 

MF-$0.76 HC-$1.95 PLUS POSTAGE 
Evaluation Methods; ^Individual Tests; ♦Military 
personnel; Models; ^Occupational Tests; *Task 
Analysis; Task Perforiance; Test Construction 
Proficiency Tests 



IDENTIFIEBS 
ABSTBACT 

A nev, sore specific language for describing work 
activities, based upon the duty lodule (clusters of tasks that tend 
to go together occupationally and organizationally in Meaningful 
ways) is being designed for the Any. ' e purpose is to iiprove 
ccsiunications between resource and reguxreient planners and prograi 
operators. The paper proposes two different kinds of lodular 
evaluation devices: sets of tasks performed by individuals, and sets 
of tasks performed by organizational units. Host of the docuient 
exasines and evaluates existing approaches to proficiency testing in 
the Arsed Forces. Sone duty nodules are listed but the proposed 
designs have been neither analyzed nor evaluated. Becosiendations are 
■ade for further developsent of the duty aodule concept. (NH) 
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INTRODUCTION 



A new, more specific language for describing work activities Is 
being designed for the Army on an experimental basis. This language is 
designed for the purpose of improving communications between resource 
and requirement planners and program operators. Also, it will permit the 
integration of future data bases. 

The need to evaluate the* feasibility of personnel Information 
systems based upon clusters of tasks smaller than a Military Occupational 
Specialty (MOS) was originally suggested by long-range planners at the 
Behavior and Systems Research Laboratory (BESRL), primarily by Mr. Cecil 
D. Johnson, who was Chief of the Statistical Research and Analysis Divi- 
sion, and Dr. J. E. Uhlaner, Technical Director. (BESRL has since 
changed its name and become part of the Army Research Institute [ARI] 
for the Behavioral and Social Sciences.) 

The development of the duty module concept was a team effort origi- 
nally conceived by ARI staff members, and developed and applied, under 
direction from ARI, by various staff members from the American Institutes 
for Research (AIR) and several consultants. Key ARI contributors Include 
Mr. Cecil D, Johnson and Mrs. Bertha H. Cory, who acted as the Contract- 
ing Officer's Technical Representatives for the Army.^ 

Key AIR staff members, in addition to Dr. Stephenson, Include Dr. 
Edwin A. Fleisbxan, Dr. Albert S. Glickman, Mr. Clifford P. Hahn, Dr. 
Ronald P. Carver, and Mr. Albert J. Farina. Key consultants at the time 
that the theoretical work was conducted Included Dr. Robert B. Miller, 
Col. Warren P. Davis (USA, Ret.), and Mr. Harry I.. Hadley. Col. Davis 
and Mr. Hadley later joined the AIR staff as project directors of the 
work currently in progress. 

2 3 

The new language is based upon a concept called the "duty module." ' 
Duty modules are clusters of tasks that tend to go together occupational ly 



Contract No. DAHC-19-71-C-0004, "A Taxonomlc Base for Future Information 
and Decision Systems," and Contract No. DAHC-19-73-C-0041, "A Comparison 
of Officer Job Content Modules with Activity Groupings Implicit in 
Course Design." 

^ Miller, R. B. A Taxonomlc Base for Future Management Information and 
Decision Systems; Theoretical Background to the Design of Duty Modules ; 
American Institutes for Research, Washington, D.C., Technical Report 
AIR-23500-7/71-TR-2, July 1971. (U.S. Army Behavior and Systems Re- 
search Laboratory, BESRL Technical Research Note, in preparation.) 

^ Stephenson, Robert W. (American Institutes for Research, Washington, 
D.C.) A Taxonomlc Base for Future Management Information and Decision 
Systems; A Conuiion Language for Resource and Requirement Planning ; U.S. 
Army Behavior and Systems Research Laboratory, Arlington, Va. , Technical 
Research Note 244 (AD-757-794) , October 1972. 
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and organizationally in meaningful ways. They are initially identified 
and formulated based upon a detailed examination task inventory or job 
analysis data. Then attention is given to ways in which these tentatively 
identified job content modules can be tested against various available 
criteria of operational utility. One relevant criterion is whether job 
content modules can be used as field asFa^'runent modules. Another possible 
application is in the area of requirement planning and unit effectiveness. 
A source of information here is data ithau can be gathered in conjunction 
with unit training and unit effectiveness exercises that are performed in 
the field.. It is of interest to determine the utility of expressing unit 
capabilities in terms of duty module classifications of tasks performed. 

The word "module" was chosen because job activity clusters, like 
equipment components of the same name, are meant to be largely self- 
contained, independent units of work. For purposes of occupational 
classification, a duty module is a cluster o^ tasks that apply without 
modification in a number of occupational classifications or specialties. 

In addition to such "individual" duty modules, modular Army training 
test components for organizational units have also been designed that 
constitute the analog of duty modules for individuals. This presentation 
will describe two different kinds of modular evaluation devices--sets of 
tasks performed by individuals, and sets of tasks performed by organi- 
zational units. 



INDIVIDUAL PROFICIENCY TESTS. 



Every individual proficiency test and every Army training test is 
already divided into special component sections with separate scores. Be- 
fore going into detail about what our modular component scores are sup- 
posed to do and supposed to look like, it is necessary to say a few words 
about what these existing r.ystems are like. We will begin with the evalu- 
ation of the proficiency of individual personnel in the Army. 

ENLISTED PERSONNEL PERFORMANCE EVALUATION 

This discussion should be prefaced by noting that evaluation of the 
performance of enlisted personnel is an important responsibility of every 
commissioned and senior noncommissioned officer in the Army. The rewards 
and punishments associated with such evaluations give commissioned and 
noncommissioned officers the necessary control over enlisted personnel to 
maintain and improve effectiveness. In addition to this important super- 
visory function, however, there are a number of formal proficiency evalu- 
ation procedures for enlisted personnel that are conducted by various 
headquarters. The mo&t important of these, for purposes of this paper, 
is the U.S. Army Enlis;:ivA Evaluation Center, located at Fort Benjamin 
Harrison, Indianapolis,' Indiana. 



The Enlisted Evaluation Center is the major operating element of the 
formal enlisted evaluation system. It was established in 195& as a Class 
2 activity of the Army. Its primary purpose then was to help the Army 
manage the proficiency pay program, which had been established response 
to the recommendations of the Cordiner Committee--the Defense \dvisory 
Committee on Professional and Technical Compensation— in 1956 and 1957. 

Monetary incentives were one of the Committee *s proposals designed 
to improve personnel retention and job motivation among trained technical 
specialties and, at the same time, stimulate higher quality performance 
among all enlisted personnel. Proficiency pay, as a concept, emanated 
from this Committee recommendation. However, an underlying principle of 
thi!> concept was that proficiency pay must be directly related to the 
demonstrated level of proficiency and must be contingent upon periodic 
checks to ensure maintenance of that proficiency. The Army enlisted 
evaluation system was developed to meet this requirement. 

ARMY ENLISTED EVALUATION SYSTEM 

The enlisted evaluation system consists of two major components: 
(a) evaluation of the enlisted man's knowledge of the various duties that 
are required at his skill level in his MOS, as indicated by MOS evaluation 
tests and performance tests; and (b) evaluation of performance in the 
currently assigned duty position, as indicated by supervisory ratings on 
the enlisted evaluation report (see Figure 1). A rating system is applied 
to the scores obtained on these instruments, and it is used to compute a 
composite score for taking individual personnel actions. This MOS evalu- 
ation score indicates the individual's relative standing among those evalu- 
ated in the same MOS and skill level and in the same pay grade. It is 
used to verify MOS qualification, to assist in determining promotion eli- 
gibility, to award proficiency pay, to guide remedial training, and in a 
variety of other personnel actions. 

PROFICIENCY TESTING "AREA SCORES" 

* 

The characteristics of the MOS proficiency testing program will not 
be detailed herein, but one particular aspect of the MOS proficiency test 
program directly relevant to this paper will be considered — the MOS "major 
area" scores. Each MOS evaluation test is organized into four to ten 
major areas; that is, four to ten subscores. The six major areas for an 
Infantry senior sergeant, for example, are weapons, tactics, field activi- 
ties, unit defense, administration, and personnel accounting. Study 
references from Army regulations, pamphlets, field and technical manuals, 
and other manuals are coded to each of the major areas in an accompanying 
test aid so that each soldier can locate the printed materials upon which 
the test is based. He may study these reference materials to improve his 
knowledge and performance. 



Promotion Zone 
of Eligibility 
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The major areas are weighted according to the relative importance 
of the functions in the missions of all units that are authorized duty 
positions in the MOS skill level i and not on the basis of the time re- 
quired to teach the subject matter in formal classroom courses nor on 
the basis of the number of personnel assigned or authorized for specific 
duty positions. The number of questions allocated to an area out of the 
total number of items in an MOS proficiency test indicates the weight 
assigned to that area, Subscores for these major areas are useful not 
only to the soldiers te^^ted, who can use the information to improve their 
performaacOi but also to various headquarters i to centralized management 
programs I and to commanders for managing assignments and training pro- 
grams • 

DIFFERENT APPROACHES TO DUTY AREAS 



Unfortunatelyi there is no consistent theoretical basis or consistent 
approach to the definition of these area scores by test developers i train- 
ing personnel I or requirement planners (see Table 1). Some of the areas 
for which the Enlisted Evaluation Center has developed scores can be 
classified as duty areas. These area scores roughly correspond to subject 
matter areas within an MOS. They are used to provide information as re- 
gards an enlisted man's strengths and weaknesses in selected subject 
areas I and they are associated with specific subject matter references. 

Table. 1. Different Approaches to Duty Areas 
Approach Duty Areas 

MOS proficiency tests Selected by test developers 

Army training schools Identified by systems 

engineering of training 

Requirement planners Associated with additional 

skill identifiers 



If subject matter references happen to be organized in terms of duty 
areas I it is easier to find the appropriate references that need to be 
studied; howeveri it is not essential. A sample list of major areas in 
an MOS is given: 

1 Weapons 

2 Tactics 

3 Field Activities 

4 Unit Defense 

5 Administration 

6 Personnel Accounting 

* 

The study guides list the regulations and technical manuals for the vari- 
ous areas > and no great amount of effort is needed to find the references 



that correspond to the area in which a low score was received in the 
proficiency evaluation test (see Table 2) ♦ 



Table 2. Sample Study Guide 



Maj or 



References 



Area 



Amy Regulations 



65-75 
210-10 



5 
6 



DA Pamphlets 
600-8 
672-2 



6 
3 



Field Manuals 
S-15 
7-10 



4 

2 



It should be made clear that we are heartily in favor of using area 
scores for proficiency tests. This was an important development in the 
design of MOS proficiency tests. Such feedback systems are an integral 
part of any sophisticated testing program. What, then, would be ques- 
tioned in the design of these area scores as they are used by the pro- 
ficiency testing system? The point raised is not 2o much how the 
proficiency testing subsystem works, but the manner in which these area 
scores interface with other personnel subsystems in the Army. 

The Army training schools, for example, identify major duty areas at 
great expense and with great difficulty, as part of their systems en- 
gineering of training process (see Figure. 2) . Systems engineering of 
training is a long, drawn-out procedure, involving detailed job analyses 
and the application of systems engineering principles and approaches in 
order to break a job into components and then select components for train- 
ing. The job is a? so organized into *'areas" when a Program of Instruction 
(PCX) is prepared. There is no consistent relationship between these POI 
area scores designed by the training people and the area scores as used 
by the proficiency training people. 

Requirement planners are also interested in a different kind of duty 
area. For example, the Army has additional skill identifiers (ASI) that 
are authorized for functional skills, which are not consistently required 
of all the job incumbents in an MOS. An example of such an additional 
skill identifier is the ability to maintain a specific type of system 
(e.g., maintenance on the Hawk Guided Missile Simulator, or the ability 
to work with specially trained scout dogs). These different approaches to 
duty areas are not necessarily incompatible with each other, but they are 
all different. Wlien you have three different parts of the same organi- 
zation--and the Army is one organization--using three completely different 
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step 1 
PERFORM JOB ANALYSIS 


1 ^ 


Identify job (Overview) 
Develop task inventory 



r 



step 2 

SELECT TASKS FOR TRAINING 



Criticality to job 
Percent Performing 
Frequency of performance 
Learning difficulty 



Step 3 

PREPARE TRAINING ANALYSIS 



Identify job conditions and standards 
Develop training objectives and criteria 
Sequence training objectives 
Identify evaluation points 



Step 4 

PREPARE TRAINING MATERIALS 




Lesson plans 
Handouts 
Schedules 
Program of Instruction 




Step 5 

DEVELOP TESTING MATERIALS 



Test outline 
Test instruments 



Step 6 
CONDUCT TRAINING 



Step 7 
PROVIDE QUALITY CONTROL 
Internal feedback 
External feedback 



Figure 2. Simplified Flow Process of Systems 
Engineering of Training.* 

From Southeastern Signal School Briefing Supplement. 
Systems engineering of training at USAS£1SS. Undated. 



language systems to describe the same kind of work, it is likely that 

there will be some unnecessary duplication of effort. This could be 

• avoided if a common language could be designed for all three parts of 

• the organization (e.g., the Army) to use. 



THE DUTY MODULE 



A duty module is a group of occupational ly interrelated tasks smaller 
than an occupational specialty. It is modular in the sense that it can 
be used as a plug-in unit to a variety of different occupational special- 
ties. Table 3 defines the module group, number, and title. Table 4 
shows an MOS duty module matrix for Army military occupational special- 
ties. As one can see, a relatively small number of duty modules can 
account for seven different MOS. Notice that each of these military 
occupational specialties has demonstrable similarity with other MOS. 



Table 3. Module Group, Number, and Title 



ADMINISTRATION 

A-1 Performs general administration at unit level 
A-2 Performs unit supervision and control of personnel 
A-3 Establishes and operates a unit m&il room 
A-4 Types, files, and performs general clerical 
operations 

TRAINING 

B-1 Conducts or participates in unit and individual 
training 

COMMUNICATIONS 

C-1 Operates unit tactical communications equipment 

(excluding use of Morse code) 
C-2 . Installs and maintains unit tactical wire 

conmunication systems 

TRANSPORTATION 

D-1 Operates unit combat support vehicles 



.(continued}. 
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Table 3 (continued) 



t 



E TACTICAL OPERATIONS 

E-1 Prepares and employs maps, charts, and instruments 

in land navigation 
E-2 Engages enemy with tank and Armor vehicle 

mounted assault weapons 
E-3 Drives tanks and associated Armor combat vehicles 
E-4 Emplaces, reports, and neutralizes tactical obstacles 
E-5 Performs in mounted, dismounted, airborne or long* 

range patrols 
E*6 Engages enemy with mortars 
E*7 Participates in ground tactical operations as 

member of a maneuver unit 
E-9 Engages enemy in close combat with individual 

weapons and machine guns 
E*10 Engages enemy with recoilless rifles and direct 

fire raissiles 
E-11 Functions under GBR warfare conditions 

F STAFF MANAGEMENT 

F-1 Performs tactical operations support duties 
F-2 Performs tactical intelligeiice support duties 

G maintenance' 

G*l Performs usei maintenance on individual and unit 
equipment and weapons (excluding motor vehicles) 
^ Performs organizational maintenance on track 
and wheel vehicle mechanical systems 

G*5 Performs organizational maintenance on track 
and wheel vehicle electrical systems 

G-4 Performs maintenance administration 

H FOOD SERVICE 

H-1 Establishes and operates a field mess 
H-2 Prepares and serves meals 
H-3 Operates a mess facility 

I SUPPLY 

1^1 Establishes and operates a unit supply point 
J PERSONNEL 

J-1 Initiates, posts, files, and retrieves information 

from personnel records 
J-2 Manages individual enlisted personnel and carries 

out manpower and personnel management programs 
J-3 Processes personal affairs actions for individuals 
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Table 4. MOS - Duty Module Matrix 



Duty 
Modules* 


IIB 


Military Occupational Specialties 
lie IID HE IIF IIG 


IIH 


A-1 






X 










A-2 


X 


X 


X 


X 




• 

X 


X 


A-3 


X 




X 


X 








B-1 


X 


X 


X 


X 


X 




X 


C-1 


X 


X 


X 


X 


X 


\ 


X 


D-1 


X 


X 


X 


X 




\ 




E-1 


X 


X 


X 


X 


X 


X 


X 










X 














X 


X 








E-4 


X 




X 


X 






X 


E-5 


X 


X 


X 


X 




X 


X 




• 


X 












E-7 


X 


X 


X 


X 


X. 


• X 


X 


E-9 


X 










X 




E-10 














X 


F-1 






X 


X 


y. 






F-2 






X 




X 







See Table 3 for definitions of duty modules. 



It is also possible to use duty modules to express personnel require* 
ments. The list of work activities in Table 5 has not been formally ap- 
proved but it illustrates the type of approach that can be used. Given 
a data processing group ^ there is a need to supervise^ to plan the analy- 
sis of the reporting, to keypunch, and so on. The number of full-time 
duty positions needed in the organization is ten. You can also specify 
the requirements in terms of the number of people qualified to perform 
each work activity. If you have a computer activity with a lot of night 
shift work, you are going to need at least three people who can supervise. 
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Table 5. Work Activity Requirements 



Work Activity 


Minimum Number of 

reopie Neeaeu 
with This Skill 


Number (or 
Proportion) of 
ruii*iime uuty 

Positions 


oupervi s ion 


• 


1 on 


Planning of analysis 
and reporting 


X 
0 




Kecdipt ana vent icat ion 
of input data 


3 


1.00 


COBOL progranuning 


2 


2.00 


Keypunching 


8 


1.50 


Computer operation i 
including peripherals . 


5 


3.00 


Interpre'ativ^n of output 


2 


i.po 


Preparation of reports 


2 * 


25 


Total Number of Work 
Activity Requirements 


30 




Minimum Number of 
People Required 


« 


10.00 



You may want one or two people for backup, in c&se of illness or vacations. 
Even though there may only be three duty positions involving supervision, 
you may want five people to be qualified as shift supervisors. Similarly, 
in planning the analysis and the reporting, you may want three qualified 
persons, but it is only a quarter- time job. In other words, defining work 
aciivitif^s in terms of duty modules can provide a more efficient use of 
personnel with all functions covered, using a minimum number of personnel. 



TESTS OF EFFECTIVENESS OF DUTY MODULES 



Resource and requirement planning experts must first agree upon the 
qualification requirements. Secondly, compatibility with work practices 
in the field is involved. For this, we can rely upon actual survey data 
regarding the way in which tasks are assigned in the field. A third test 
is to evaluate the usefulness of the module in planning and evaluating 
the requirements for and performance of organizational units. 
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TEST 1 

A firbt test is to ask experts to design job content modules. Typi- 
cal job content modules are: 

(1) Operates unit tactical communications equipment, 
excluding Morse code. 

(2) Installs and maintains unit tactical wire communi- 
cation system. 

In this case, the people who are asked to operate unit tactical communi- 
cations equipment are usually different from those who install and 
maintain it. Neither of these work activities, however, is a full-^ime 
position for anyone. These are modular things that can be assigned to 
different people. 

TEST 2 

A second test of duty modules is compatibility with assignment 
practices in the field. Some data have already been analyzed (see Figure 
3). Data were used that were already in existence, and that had been 
collected with task inventories that were administered by the Army Office 
of Personnel Operations. The data base is called the Military Occupations 
Data Bank, or MODB. The original task statements in MODE are organized 
in terms of functional areas of performance (see the administration and 
training columns on the right side of Figure 3). the rows correspond to 
task clusters that were identified in an empirical clustering of tasks. 




Figure 3. Test 2, Compatibility with Assignment 
Practices In the Field (In Percent). 




# 



The Comprehensive Occupational Data Analysis Programs (CODAP) 
system, developed by the Air Foj.xe, was used. This system was developed 
by Ray Christal, Joe Ward, Bill Lecznar, and others at the Air Force 
Personnel Laboratory. People were not clustered together on the basis 
of their similarity in terms of task performance, which is the way this 
program is usually used for clustering purposes. Instead, tasks were 
clustered together on the basis of the probability that the tasks would 
be assigned to the same people. On the left side of Figure 3, cluster- 
ing is compared with some duty modules that were analyzed in terms of 
this particular CODAP run (see Table 3 for definitions of duty modules) . 
The computer run suggested that there were seven task clusters in this 
particular group of tasks. The percentages shorn in the duty module 
columns on the left side of Figure 3 indicate the percent of tasks in 
each of the empirically identified clusters that fall into duty module 
categories A-1 through B-2. The percentages shown on the right-hand side 
of the figure indicate the percent of tasks in the empirically identified 
clusters that fall into each of the administrative areas used to group 
tasks in the Military Occupations Data Bank. 

It is important to note that the design of the duty modules is not 
complete, nor is the preparation of the task statements. These task 
statements will be revised to reflect the computer output. In due course, 
the definition of our duty modules will correspond with the assignment 
practices that actually exist in the field for any given specialty. 
Thus, the clusters will correspond more closely with the duty modules. 

It does not necessarily follow that there will be complete agreement 
with the computer runs. For example, it is possible that the first three 
empirical task clusters would be considered as really one cluster rather 
than three. Before that conclusion can be made, many other occupational 
specialties, in addition to the one that these data are based upon, will 
have to be reviewed. Essentially, duty modules tre derived from many 
different occupational specialties rather than just one, which is the 
case with this particular computer run. The decision as to whether these 
first three clusters are one cluster or two or three clusters must be 
based upon data in other occupational specialties as well. 



TEST 3 

A third test of duty modules is the relationship to unit performance 
in the field. Task checklist items have been devised based upon our duty 
modules. Each duty module consists of a collection of task iitatements. 
We were thus able to study Army Training Tests (ATTs) and convert task 
statements into checklist items. This provides the mea\s by which infor- 
mation can be collected about the performarco of individuals during unit 
training tests. Separate scores are obtained for separate practice maneu- 
vers. Separate scores can be obtained for movement, defense, and attack, 
as noted in Figure 4. Each exercise will have a scenario, the possibility 
of casualties, and so forth. Duty modules will be looked at during the 
appropriate phase of the Army Training Tests. They would not be tested 
during every single phase. 

- 13 - 
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Unit Test Phase 



Module 



Task Checklist Item 



Movement 



Defense 



Attack 



C-1 



Establish and operate 
field communications 
relay station 



X 



G-1 



Make entries in 
equipment log books 



X 



X 



Figure 4. Step 3, Relationship to Unit Performance. 



SUMMARY 



These individual duty modules are designed to be derived from a 
variety of specialties rather than just one. They are economical in the 
sense that many different task inventories can be designed with a small 
set of duty modules. They utilize task inventory data already collected 
cn assignment and work assignment practices in the field. They are de- 
signed to improve communications between resource planners and require- 
ment planners. They meet an apparent need for consistency at a level of 
generality between the MOS and the task, at an optimum level of detail, 
and they permit more precise matching of personnel to job requirements. 



It is not possible to talk about testing in terms of duty modules 
without first talking about testing standards. Performance standards in 
the Army are prepared systematically as part of the systems engineering 
of training process for the design of Army school courses. At one point 
in systems engineering, task and skill analysis sheets are prepared. 
These result in evaluation plans. It is a standard practice to indicate 
specific performance standards for training purposes for each terminal or 
facilitating objective on these evaluation plans (see Figure 5). 

The evaluation plans only cover those areas of interest to the 
scliool, however. They usually do not cover all the tasks in an MOS, es- 
pecially for the more advanced tasks, which the personnel are supposed to 
learn on the job. The Enlisted Evaluation Center, therefore, has to sup- 
plement this school-oriented information with other sources. Usually, 
they use the judgments of knowledgeable NCOs who have experience in the 
particular MOS, and who formulate proposals regarding what tasks are 
appropriately included in an MOS proficiency test. 



PERFORMANCE STANDARDS AND SKILL LEVELS 
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1. Criteria for the Training Objective developed for the task; 



Action 



Troubleshoot AN/TRC-24. (A-22-1) 



Condition 



In addition to Standard Training Conditions, 
the student is given an AN/TRC-24 with one 
major component containing a DS part defect 
as well as*0S-8, ME-30/U, TM 11-5820-287-12, 
TM 11-5820-287-34, and AN/TRC-24 Block 
Diagram. 



Standard 



The student is qualified if, when given two 
defective AN/TRC-24s with a 2-hour time limit 
on each, he can isolate one of the defective 
parts. 



Figure 5. Evaluation Planning Infoimatlon Sheet. 



The proficiency testing system in the Army, as it is presently orga- 
nized, provides separate tests for each skill level within each MOS. 
Occasionally, one test may be used in two or three skill levels with 
different score requirements, but the principle is the same. One reason 
frr providing separate tests for different skill levels is that an MOS is 
a broad collection of duty areas that cover many different duty positions. 
Providing separate tests for each skill level makes it possible to pro- 
vide items that are more appropriate for the positions being filled by 
those who take the tests. 

This skill level approach, which is not incompatible with the duty 
module concept that we have described, is appropriately used in connec- 
tion with duty modules, and, further, it illustrates how useful duty 
modules can be to those who design tests. Table 6 lists the tasks for a 
duty module of patrolling, either mounted or dismounted. One's skill 
level is dependent upon whether one supervises a task, does the task and 
also supervises it, simply does the task, or whether one just assists in 
in doing it. The three skill levels in the IIB MOS (Light Weapons Infan- 
tryman) are indicated. The skill levels are numbered 1, 2, and 4. There 
are only three skill levels in this particular MOS, so there is no skill 
level numbered 3. Table 6 indicates that people who are at skill level 4 
are more likely to supervise. People who are at skill level 2 do not 
supervise, and arc much more likely to assist somebody. People who are 
at skill level 1 carry out the orders and requirements of their superiors. 
This kind of information about skill level profiles could be extremely 
useful to Army organizations in designing proficiency tests. 
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Table 6. Duty Module E-5: Patrols, Either Mounted or Dismounted 



Tasks 

(1) Plan patrol operations 

(2) Assemble, inspect, issue patrol 

order, and lead patrol 

(3) Operate listening or 

observation post 

(4) Serve in combat patrols 

(5) Serve in reconnaissance patrols 

(6) Serve in ambush patrols 

(7) Mark route or serve 

as guide for unit 

(8) Participate in air search 

operations or air delivered 
patrol 

(9) Estimate charge, emplace and 

fire demolitions 



Performance Expectations for 
Skill Levels 1, 2 and 4 of IIB MPS 

Super- Do and 

vise Supervise Do Assist 

4 2 



a 



4 
4 



4 
4 



1 

2. 1 
2, 1 

2, 1 
2, 1 

2, 1 
2, 1 



•There are only three skill levels in the IIB MOS: 1, 2 and 4. 

^Task No. 7 (when performed by an 1 IB MOS) is normally supervised by 
someone in another MOS. 



COST ADVANTAGES OF A MODULAR APPROACH 



Previously mentioned were the ways in which the Army military occu- 
pational specialties were modular. Refer to Table 4 and note that with 
this matrix one can account for many different tasks with a relatively 
small number of duty modules. Moreover, this matrix is just the corner of 
a much bigger matrix. In our present work with enlisted duty modules, we 
have developed 31 duty modules that can account completely for 16 differ- 
ent MOS. 
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To appreciate the possible savings is to translate this information 
on duty modules and MQS into test items. Say that these 16 MOS would re- 
quire 100 items apiece to account for them if they were developed inde- 
pendently by various test -developing agencies. That would make 1,600 
test items I if you used an independent approach to test development. We 
estimate that it takes only 10 items apiece to describe a duty module. 
In other words, it is possible that 310 items can do essentially the same 
job as 1,600 test items. To be able to prepare 310 items rather than 
1,600 items reflects a considerable savings. It is contingent upon de- 
fining modules that cut across, and have the same meaning in, different 
occupational specialties. In giving these figures, we have not discussed 
skill levels; but different tests for different skill levels would be re- 
quired in both systems. Thus, multiply the number of items in our ex- 
ample by three or four to get the number of test items that would actually 
be needed by the people who design these tests and work with them. 



MODULAR ARMY TRAINING TESTS (AHS) 



A.t the present time, the Army has several hundred Army Training Tests 
(ATTs) for use in evaluating the performance of organizational units. 
Each of these tests has a scenario and provisions for referees who are 
trained to follow people and take notes as regards their performance in 
the unit test. The question to be posed here is this: Would it be de- 
sirable for Army Training Tests for organizational units to be organized 
in the same modular fashion that we have proposed for individual pro- 
ficiency tests? 



PURPOSE OF ORGANIZATIONAL UNITS 

Before further <Iiscussion of the feasibility of modular ATTs, the 
accuracy and consistency with which the intended purposes of organization- 
al units have been specified must be considered. The Army has several 
different terms for describing the intended purposes of organizational 
units: a primary mission, some functions, and a capability. 

A primary mission is defined as the principal purpose that an organi- 
zation is designed to accomplish. The functions are the appropriate or 
assigned duties, responsibilities, missions, or tasks of an individual 
office or organization. A capability is the ability to execute a speci- 
fied course of action. Further details will not be discussed except to 
state that after studying the various mission, functional, and capability 
statements in Army documents, it was concluded that the capability state- 
ment was the one that should be used as the basis for structuring organi- 
zational unit testing modules. 
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THE CAPABILITY STATEMENT 



A possible capability statement for a theoretical unit is shown in 
Table 7. It is clearly possible to analyze a capability in terms of spe- 
cific component functions and operational criteria, as shown in the table. 



Table 7. Theoretical Capability of a Unit 

1. Title: 

Transport supplies and resupply itself. 

2. Essential Component Functions: 

a. Load, move, and unload unit loads of rations, 
POL, ammo, and repair parts. 

b. Repair minor vehicular failures enroute. 

c. If unable to make minor vehicular repairs > tow 
inoperable vehicles. 

d. Move unit loads on the road or cross-country. 

e. Pick up and issue supplies. 

3. Minimum Operational Criteria: 

a. Sufficient vehicles on hand in condition to 
move unit load. ^ 

b. Sufficient vehicles on hand in condition to 
pick up and deliver supplies. 

c. Trained drivers. 

d. Authorized maps and compasses on hand. 

e. Supply platoon trained as a team. 

f. Trained supply personnel. 

g. Trained vehicular and radio mechanics. 

h. Satisfactory status of equipment maintenance. 

i. Satisfactory completion of ATT and FTX. 

4. Standards (To be developed): 

Includes minimum personnel, skills, operable 
equipment, and training necessary to be con- 
sidered C-1, C-2, or C-3 as defined in AR 220-1. 
Standards below C-3 are C-4. 



The capability of the unit is to transport supplies and to resupply it- 
self. What is involved in this capability are the functions of loading, 
repairing vehicles, picking up initial supplies, and so forth. The mini 
mum operational criteria are also indicated. A Department of the Army 
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study of output measurement conducted in 1968 shows the same conclusion 
that we express--that capabilities are a good way to structure and 
organize ATTs. 

CRITERIA FOR MODULAR ATT EVALUATION DEVICES 

We have formulated a number of criteria for an improved system of 
evaluating performance in Army Training Tests. Details are beyond the 
scope of this presentation, but the criteria and related recommendations 
are given. 

• A series of clear, quantitative statements specifying the 
capabilities of a unit. 

• A taxonomy of unit capability statements. . If possible, 
these statements should be modular. 

• Criteria of unit effectiveness based upon and relatable 
to the taxonomy of unit capabilities, and consistent with 
and relatable to criteria for the performance of indi- 
viduals in the unit. 

• Both kinds of criteria (individual and unit) based upon 
performance standards rather than relative standing in 
test performance. 

« 

• Varied performance standards depending upon situational 
conditions, such as terrain, percent casualties, and 
resource inputs. 

• Aggregation of standards for organizational components and 
generation of an overall index for the unit as a whole. 

• The emphasis in criteria statements upon end results 
rather than methods used to achieve the results. 

• Output measures allowing for the possibility in evaluations 
of corrective measures that may have been -aken by command 
personnel, and that would permit the unit to meet standards 
in spite of some departure from expected procedures. 

• Scoring that provides specific information about the 
leadership of a unit. 



Department of the Army. Improvement of Output Measurement , Report of 
a Special Study by the Army Staff Coordinated by the Comptroller of the 
Army, January 1968. 



ERIC 



. 19 - 

21 



• Scoring that provides comprehensive evaluations of the 
unit when tested as an entire unit. 

• Quantitative weights assigned for satisfactory performance 
. and deducted for inadequate performance that reflect the 

probable seriousness of the actions. 

• A method for relating output measures to input measures 

• logically. 

• ■ 

SUMMARY . 

A series of clear, quantitative statements are needed to specify 
the capabilities of a unit. There is also a need for a taxonomy of unit 
capability statements. A taxonomy is a theoretically-based language . 
that implicitly classifies or categorizes a capability statement at the 
same time that it describes it. 

♦ 

Capabilities should also be modular in nature; that is, the capa- 
bility statement for an Infantry battalion should be the same, if pos- 
sible, as a highly similar, closely related capability statement for an 
Armored Cavalry squadron. If you can design the capability statements 
thusly, and organize the testing accordingly, it is possible to design 
one modular unit training test component that would be useful for In- 
fantry battalions and for Armored Cavalry squadrons. Examples of possi- 
ble modules are night ground attack, retrograde movement, and stationary 
defense. Since many different capabilities of different kinds of bat- 
talions are common, having similar tests provide many economic features. 



CURRENT WORK ON MODULAR ATTS 



Having proposed the design of modular Army Training Tests, we pro- 
ceeded to design some. Some field survey work was conducted in conjunc- 
tion with Army Training Tests at Fort Lewis, Washington, in August 1973, 
to pre-test the first versions of these modular evaluation devices. 
These devic^is and data have not been analyzed or evaluated as yet, thus 
our comments arc based upon what was learned during the design stages. 



PERSONNEL CAPABILITIES 

In designing these evaluation devices, two things quickly became 
apparent: (a) devices could not be designed for units based upon en- 
listed duty modules alone; and (b) there was a need for officer duty 
nodules. The Army also needed information about officer jobs and had 
contracted for job analyses of about 200 officer jobs. These job analy- 
ses were used as the basis for designing initial duty modules for officer 
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jobs. Table 8 defines officer duty modules by area and module number. 
Table 9 accounts for all the major duties of an officer's job with just 
a few duty modules. Each of the positions shown is completely accounted 
for by the officer duty modules listed. (The letter refers to the 
"areas" from which the module was taken in Table 8, and the number 
identifies a specific module within that area.) 



Table 8. Officer Duty Modules by Area 



Ntmber. of 

Area Title Modules 

A Command Management, General Management 

and Administration 9 

B Personnel 4 

C Intelligence 5 

D Operations and Plans (Staff) 4 

E Organization, Training 3 

F Logistics (Staff and Consumer Units) 9 

G Communications and Electronics 2 

H Civil-Military Affairs 3 

I Comptroller ship. Budget and Fiscal 2 

J Army Aviation 5 

K Research, Development^ Test and Evaluation 2 

L Operations Research and Systems Analysis 1 

M ADP Management and Programrtiing 1 

N Educatio:), Inswruction 2 

0 Information Activities 1 

U Tactical Direction of Combat Units S 

W Miscellaneous 9 

X Individual Functions and Special Qualifiers 4 

. FF Logistical Services 9 

HH Supply and Maintenance Support Operations 9 
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Table 9. Application of Duty Modules to Officer Positions 



Position 

Cdr., Infantry Rifle Co. 
CPT 

Cdr., Reception Station Co. 
LT 



Asst. Army Attache 
LTC 



Duty Modules 

A-1, A-3, A-4, A-6, A-8, 
E-1, F-1, X-1, X-2 

A-1, A-3, A-5, F-1 
A-1, A-4, C-4 



EQUIPMENT CAPABILITIES 

In addition to information on the personnel capabilities of a unit, 
there was a need for detailed information about the capabilities of 
equipment. The need to think about equipment capabilities became ap- 
parent when preparing a unit capability table, which described different 
types of capability for each component in a platoon (Table 10) . 



Table 10. Equipment Capability Table for Armored 
Cavalry Platoon (TOE 17-107H) 



1. 



Item of 
Equipment 

Antenna (AT- 
784/PRC) 



2. Armored Re- 
connais- 
sance 
Airborne 
Assault 
Vehicle 
(M 551) 



Basis of Issue 

2-Scout Section 
1-Rifle Squad 



3-Light Armor 
Section 



Capability 

Determine the direction 
to a specific radio 
transmitting in the 
frequency range 30.0 
to 75.95 MHz. 

Negotiate almost any 
terrain at speeds 
from 4 miles per hour 
in water to 43 mph on 
roads, including 7- 
foot spans, 33-inch 
vertical obstacles 
and 60% grades. 



Reference 



ST-24-18-1 



ST-17-1-1; 
ST-17-15-1 
FM-17-36 
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It was not possible to describe the fire-power capabilities of a 
squad that was equipped with a certain type of machine gun, for example, 
without knowing what type of machine gun it was. One machine gun might 
be capable of a sustained firing of 40 rounds per minute, while another 
might fire 100 rounds per minute. The range of the first machine gun 
might be 6,000 meters, while the range of the second, faster machine gun 
might.be 3,000 meters. Clearly, any quantitative statement of the capa- 
bilities of that squad, and hence the capabilities of the whole platoon, 
is greatly affected by which of the two machine guns is being used. 
Similar constraints upon unit capability statements are imposed by the 
kind of transportation that is available. One vehicle, for example, is 
capable of a road speed of 40 miles per hour, a water speed of 3 miles 
per hour, and has a range of 300 miles, a cargo space of 23 cubic feet, 
and can carry a cargo pay load of 3,000 pounds, including the driver. 
When carrying troops, this vehicle is limited to the vehicle driver, the 
commander, and eleven passengers. This type of information has obvious 
implications as regards the capabilities of a unit. 



PRELIMINARY WORK ON EVALUATION DEVICES FOR AHS 

As stated earlier, one of the requirements for the individual en- 
listed modules is that they be meaningful in terms of unit evaluation 
procedures as well as individual evaluation procedures. Consequently, 
we designed Army Training Tests that were not solely modulp.r in the sense 
that they could be useful in evaluating different kinds of Army units. 
They were also composed of scoring evaluation procedures for individuals, 
based upon performance in duty module terms. 



APPROACH TO THE PROBLEM 

The first approach was to develop examples of criterion behavior for 
a particular duty module, but it did not seem to be workable because too 
much territory was covered with one global rating. 

A new approach was developed. Since each duty module is defined by 
approximately ten task statements, we decided to design our evaluation 
devices in terms of checklist items based upon these task statements 
(Figure 6) . The method of grouping the items was retained so that the 
devices were still organized in terms of duty modules. The overall per- 
formance of the unit could not be scored exclusively in terms of indi- 
vidual performance, however. Some unit performance indices were 
developed. An example of overall unit performance rating procedures 
is shown in Figure 7. 



ERIC 



- 23 - 



as 



ATT Phases 



Module Number 


• 

Checklist Item 


Movement to and 
Occupation of 
Forward Assembly Area 


Reconnaissance 


Covering Force 

I . 


Route and Area Security 


Flank Guard and 
, Economy of Force 


Overall Rating 


A-2 


1. 


Replacements properly 


















received and assigned 
















2. 


Losses and casualties 


















properly processed 
















3. 


Leadership of platoon 


















and squad NCOs 
















4. 


Duties of subordinates 


















properly allocated 















Figure 6. An Approach to Unit Evaluation Devices 
Based upon Task Statements. 



Activity Rated: Rifle Platoon 


Score 


Comments 


Phase 1— Da/light Attack 






Proper actions and preparations in 
assembly area? 

Proper organization, formation, and 
dispersal? 

Platoon *s use of cover and concealment? 

Firing on objective--g od voJume, well 
directed? (Scored only for live fii^ing) 



Figure 7. Evaluation In Terms of Unit Performance As a Whole. 
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RECOMMENDED APPROACHES AND PLANS 



APPROACH 

Our modular approach is summarized in the following description. 
Tests have usually been designed by taking a particular criterion situ- 
ation and designing a test for that one situation irrespective of how it 
is structured for that particular case at that particular point in time. 
If, for example, a proficiency knowledge test is being designed for 
iomebody who repairs automobiles, you determine exactly which tasks 
people in that MOS are supposed to perform, then you design the indi- 
vidual proficiency tests to measure knowledge of those specific tasks. 

The same is true of unit training tests. You look at the capability 
statements, study the terrain in which the test, is to be conducted, and 
then you design some tests for that particular type of unit in that par- 
ticular situation. If you are dealing with an Infantry unit, you design 
an Infantry test; if you are dealing with an Armored Cavalry unit, you 
design an Aruiored Cavalry test. There is relatively little overlap or 
systematic utilization of materials prepared by other people for differ- 
ent kinds of units. 

What we are proposing for individual testing is that personnel ex- 
perts in the various services spend a lot more time dealing with a vari- 
ety of occupational specialties rather than focusing upon one specialty 
at a time. If possible, they should change the way in which occupational 
specialties are defined, and define different occupational specialties 
in terms of a common set of duty modules. 

What we are proposing for organizational units is that evaluation 
devices be designed so that different kinds of organizational units can 
be evaluated with the same kind of modular evaluation devices. The modu- 
lar approach, in both cases, facilitates feedback regarding performance 
and makes the feedback to the individual or unit being tested much more 
meaningful. Both strong points and weak points are much better defined. 

It is possible to argue that feedback about test performance should 
not' be too specific. The Air Force, for example, does not provide infor- 
mation about area scores in its Skill and Knowledge Tests (SKTs) , which 
are its equivalent of the Army's proficiency test. The reason given is 
that people would study for the test rather than study for the job (i.e., 
the Air Force would seem to be worried about the unintended consequences 
of providing information in terms of proficiency testing a^ea scores) . 
Available data suggest thr.t area scores have been successful in the Army, 
however, and the unintended consequences simply have not occurred. 

It is our opinion that, regardless of whether you provide feedback 
in terms of the modular scores that we have described, or whether you 
provide feedback in terms of areas defined by a committee of NCOs or unit 
commanders, area scores arc useful feedback and should be used. 
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Our proposal that the Army develop modular unit tests becomes some- 
what complicated because the Army has already embarked upon a large- 
siale program to overhaul all of its unit training tests. The Army 
recently designed a systems engineering of unit training programs that 
is patterned after its systems engineering of training procedures for 
individual training. Several organizations are currently working on 
the redesign of unit training programs in terms of the Army's systems 
engineering of unit training programs. Thus, a number of new evaluation 
devices will definitely be designed. We have discussed our approaches 
to modular unit training tests with those who are responsible for this 
type of work, and we hope that they will consider a modular approach to 
ATTs as an alternative when the tests are revised. 

USES OF DUTY MODULES 

We have stressed the economy of duty modules in terms of test 
preparation costs and the importance of consistency in language. Other 
possible advantages arc given. 

(1) Duty modules can improve occupational research, its 
description, and utilization. 

(2) They have the potential to reduce training time and 
lower training costs. 

(3) They can provide a better use of individuals in 
assignment substitutions. • 

(4) They can siinplify automated assignment and control 
procedures . 

(5) They can improve proficiency evaluation. 

(6) They can improve career guidance and planning. 

(7) They can improve utilization of personnel at a 
local level. 

(8) ihey can improve unit training evaluations. 



PRESENT IMPLEMENTATION OF PROCEDURES 

It will take many years before the kind of modular system that we 
propose can be established. If everyone agreed that it should be done 
tomorrow, it would still take several years before such a system could 
become operational. Nevertheless, we think that the advantages of a 
nodular approach to test construction are so great that people should 
start thinking about it now. The problem arises because test developers 
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cannot proceed by themselves. The whole occupational structure needs to 
be revised so that occupational specialties would be defined in terms of 
duty modules. 

Nevertheless, there are several procedures that could be started 
now. For example, change the way in which test design work groups are 
established. Instead of assigning test development to a group of ex- 
perts in the same specialty, credte a work group comprising representa- 
tives from several specialties, and ask that group to design a single 
test component that would be useful to them all. A similar approach 
(i.e., assigning groups of experts from different types of units) can be 
used with those who design unit evaluation devices. These preliminary 
steps, taken now, would have many immediate advantages, and would great- 
ly facilitate a conversion to modular testing in the future. 

• 

FUTURE PLANS 

Our current work on the design of modular personnel systems is 
moving in the direction of skills inventories, improved assignment sys- 
tems, and the design of career-progression patterns . Special interest 
Is in career-progression patterns that have some kind of optimum dis- 
crepancy between present duty module qualifications and the duty module 
qualifications needed for the next assignment. 

Several other modular personnel systems are also in the planning 
stage, because the flexibility provided by a modular system makes many 
things possible that were previously out of the question. This widening 
of possibilities becomer exciting, and the work is fun~we look forward 
to its many challenges. 
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